{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Alt Text](./APP.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Eval\n",
    "There are broadly 2 types of evaluations:\n",
    "1. Offline\n",
    "2. Online\n",
    "\n",
    "\n",
    "## Offline Evaluation\n",
    "This is where you build the models, eval models, LLMs setc and you mostly compute everything in a controlled environment. This is done against a set benchmark, GT answers, Reference contexts etc. Hit Rate, NDCG, MRR, Precision@K, Recall@K etc etc are used here to check the effectiveness of Embedding Model + Vector DB + Rerankers + Chunking Strategy \n",
    "\n",
    "\n",
    "On the other hand: Fluency, Complexity, Perplexity, BERTScore, BLEU, ROUGE, METROR, LLMasJudge, Groundedness, Hallucination Rate, Toxicity, Context Adherence, Faithfulness etc etc are used to judge the quality of LLM and it's response. You can use your rubric fine tuned models to use them as LLMasJudge.\n",
    "\n",
    "## Online Evaluation\n",
    "This is where we do live evaluation. No ground truth and there is `pipelines` in place having all the components to figure out shortcomings of system. These are multiple `Choke / Failure Points`. For example:\n",
    "\n",
    "1. LLM itself is slow, bad, toxic, bias etc and prone to injections etc. It may be prone to divulging sensitive info\n",
    "2. Context retrieval models are slow or not good enough so you need to use different chunk size, different chunking strategy etc\n",
    "3. VectorDB uses ANN isntead of pure Cosine similarity so it has it's issues. So you end up using Re-Ranksers. On the other hand, not all tasks need sementic search so you need to use Syntactic search\n",
    "4. API failure rate, Latency, Time to response, throughput, load resistance etc etc are checked here\n",
    "\n",
    "\n",
    "\n",
    "## Types of Eaaluation metrics:\n",
    "In this repo, you'll find 6 different types of Classes where 4 of them are actively working and 2 are abstract classes. The 5 working classes make up for more than 50+ metrics that are used. These are as follows:\n",
    "\n",
    "### `IOGuards`\n",
    "Guads to protext the model from taking in prompt injections, divulging sensitive data bias, toxicity, polarity, harmful output, sentiment etc etc for Query, Context and Response\n",
    "\n",
    "### `TextStat`\n",
    "These are mostly for Response answering the questions like:  How complex is the output, how understandable is it, how fluent, which calls of student can understand it etc etc\n",
    "\n",
    "### `ComparisonMetrics`\n",
    "Mostly used for Summarisation\n",
    "\n",
    "These are Query-Context, Query-Response and Context-Response based metrics. They tell you the answer quality according to query and response. Metrics are Hallucination, Contradiction, BERTScore, ROUGE, METEOR etc etc\n",
    "\n",
    "Within it, there is string matching based ones using BM25, Levenshtein Distance, Fuzzy Score etc (Need to add LSH and other Hashing too)\n",
    "\n",
    "### `AppMetrics`\n",
    "Checks the overall APP usage. Failure rate, latenct, time to response, time to fetch the context, time for LLM to answer etc etc. Newrelic, Prometheus etc are there which be directly integrated with  Flask or FastAPI. Streamlit is not supported by them but I have written by own decorstors for time.\n",
    "\n",
    "### `LLMasJudge`\n",
    "These are nothing but a wrapper. You use a prompt and send the question, answer and context the LLM to get response. If it is offline evaluation, you can get reasoning steps and compare them against GT. You can as for reasonong for answering and compare against GT, Humans. This can be used for any task. Many models like Prometheus-2, PHUDGE, JudgeLM etc are there finetuned for specific tasks\n",
    "\n",
    "### `TraditionalPipelines`\n",
    "You use traditional pipelines for storing the topics from Query, Context, Response to evaluate and compare whether they all talk about same thing or not. Then you can use POS tagging and other classification tasks for your usecase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Requirements\n",
    "Tested with: `Python 3.9`\n",
    "\n",
    "Step: \n",
    "1. `pip install -r requirements.txt`\n",
    "2. `pip install -U evaluate` (without it, some old metrics won't work)\n",
    "3. `streamlit run eval_rag_app.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the below code will run the `st.spinner` while loading models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package vader_lexicon to\n",
      "[nltk_data]     /Users/mahkumar/nltk_data...\n",
      "[nltk_data]   Package vader_lexicon is already up-to-date!\n",
      "Some layers from the model checkpoint at d4data/bias-detection-model were not used when initializing TFDistilBertForSequenceClassification: ['dropout_19']\n",
      "- This IS expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some layers of TFDistilBertForSequenceClassification were not initialized from the model checkpoint at d4data/bias-detection-model and are newly initialized: ['dropout_39']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "/Users/mahkumar/anaconda3/envs/py39/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n",
      "WARNING:evaluate_modules.metrics.evaluate-metric--bleurt.98e148b2f8c4a88aba5037e4e0e90c9fd9ec35dc37a054ded8cfef0fa801ffab.bleurt:Using default BLEURT-Base checkpoint for sequence maximum length 128. You can use a bigger model for better results with e.g.: evaluate.load('bleurt', 'bleurt-large-512').\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reading checkpoint /Users/mahkumar/.cache/huggingface/metrics/bleurt/default/downloads/extracted/3ab93262e863625b5602d5c988317eca1e3022de221c7e6e9b88b58fca9ee841/bleurt-base-128.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Reading checkpoint /Users/mahkumar/.cache/huggingface/metrics/bleurt/default/downloads/extracted/3ab93262e863625b5602d5c988317eca1e3022de221c7e6e9b88b58fca9ee841/bleurt-base-128.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Config file found, reading.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Config file found, reading.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Will load checkpoint bert_custom\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Will load checkpoint bert_custom\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Loads full paths and checks that files exists.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Loads full paths and checks that files exists.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... name:bert_custom\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... name:bert_custom\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... vocab_file:vocab.txt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... vocab_file:vocab.txt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... bert_config_file:bert_config.json\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... bert_config_file:bert_config.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... do_lower_case:True\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... do_lower_case:True\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... max_seq_length:128\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:... max_seq_length:128\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Creating BLEURT scorer.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Creating BLEURT scorer.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Creating WordPiece tokenizer.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Creating WordPiece tokenizer.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:WordPiece tokenizer instantiated.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:WordPiece tokenizer instantiated.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Creating Eager Mode predictor.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Creating Eager Mode predictor.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Loading model.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Loading model.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:BLEURT initialized.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:BLEURT initialized.\n",
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     /Users/mahkumar/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to /Users/mahkumar/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package omw-1.4 to\n",
      "[nltk_data]     /Users/mahkumar/nltk_data...\n",
      "[nltk_data]   Package omw-1.4 is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "from eval_metrics import *\n",
    "\n",
    "guard = IOGuards()\n",
    "stat = TextStat()\n",
    "comp = ComparisonMetrics()\n",
    "\n",
    "\n",
    "def evaluate_all(query, context_lis, response):\n",
    "    context = \"\\n\".join(context_lis)\n",
    "\n",
    "    RESULT = {}\n",
    "\n",
    "    RESULT[\"guards\"] = {\n",
    "        \"query_injection\": guard.prompt_injection_classif(query),\n",
    "        \"context_injection\": guard.prompt_injection_classif(context),\n",
    "        \"query_bias\": guard.bias(query),\n",
    "        \"context_bias\": guard.bias(context),\n",
    "        \"response_bias\": guard.bias(response),\n",
    "        \"query_regex\": guard.detect_pattern(query),\n",
    "        \"context_regex\": guard.detect_pattern(context),\n",
    "        \"response_regex\": guard.detect_pattern(response),\n",
    "        \"query_toxicity\": guard.toxicity(query),\n",
    "        \"context_toxicity\": guard.toxicity(context),\n",
    "        \"response_toxicity\": guard.toxicity(response),\n",
    "        \"query_sentiment\": guard.sentiment(query),\n",
    "        \"query_polarity\": guard.polarity(query),\n",
    "        \"context_polarity\": guard.polarity(context),\n",
    "        \"response_polarity\": guard.polarity(response),\n",
    "        \"query_response_hallucination\": comp.hallucinations(query, response),\n",
    "        \"context_response_hallucination\": comp.hallucinations(context, response),\n",
    "        \"query_response_hallucination\": comp.contradiction(query, response),\n",
    "        \"context_response_hallucination\": comp.contradiction(context, response),\n",
    "    }\n",
    "\n",
    "    RESULT[\"guards\"].update(guard.harmful_refusal_guards(query, context, response))\n",
    "\n",
    "    tmp = {}\n",
    "    for key, val in comp.ref_focussed_metrics(query, response).items():\n",
    "        tmp[f\"query_response_{key}\"] = val\n",
    "\n",
    "    for key, val in comp.ref_focussed_metrics(context, response).items():\n",
    "        tmp[f\"context_response_{key}\"] = val\n",
    "\n",
    "    RESULT[\"reference_based_metrics\"] = tmp\n",
    "\n",
    "    tmp = {}\n",
    "    for key, val in comp.string_similarity(query, response).items():\n",
    "        tmp[f\"query_response_{key}\"] = val\n",
    "\n",
    "    for key, val in comp.string_similarity(context, response).items():\n",
    "        tmp[f\"context_response_{key}\"] = val\n",
    "\n",
    "    RESULT[\"string_similarities\"] = tmp\n",
    "\n",
    "    tmp = {}\n",
    "    for key, val in stat.calculate_text_stat(response).items():\n",
    "        tmp[f\"result_{key}\"] = val\n",
    "    RESULT[\"response_text_stats\"] = tmp\n",
    "\n",
    "    return RESULT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of RobertaModel were not initialized from the model checkpoint at roberta-large and are newly initialized: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'guards': {'query_injection': [{'label': 'SAFE',\n",
       "    'score': 0.9999986886978149}],\n",
       "  'context_injection': [{'label': 'SAFE', 'score': 0.9999991655349731}],\n",
       "  'query_bias': [{'label': 'Biased', 'score': 0.6330747604370117}],\n",
       "  'context_bias': [{'label': 'Non-biased', 'score': 0.5858706831932068}],\n",
       "  'response_bias': [{'label': 'Biased', 'score': 0.5588837265968323}],\n",
       "  'query_regex': {},\n",
       "  'context_regex': {},\n",
       "  'response_regex': {},\n",
       "  'query_toxicity': [{'label': 'toxic', 'score': 0.9225953817367554}],\n",
       "  'context_toxicity': [{'label': 'toxic', 'score': 0.9640267491340637}],\n",
       "  'response_toxicity': [{'label': 'non-toxic', 'score': 0.9988303780555725}],\n",
       "  'query_sentiment': {'neg': 0.701,\n",
       "   'neu': 0.299,\n",
       "   'pos': 0.0,\n",
       "   'compound': -0.6908},\n",
       "  'query_polarity': [{'negative': 0.98,\n",
       "    'other': 0.01,\n",
       "    'neutral': 0.01,\n",
       "    'positive': 0.0}],\n",
       "  'context_polarity': [{'negative': 0.96,\n",
       "    'other': 0.03,\n",
       "    'neutral': 0.01,\n",
       "    'positive': 0.0}],\n",
       "  'response_polarity': [{'negative': 0.7,\n",
       "    'other': 0.19,\n",
       "    'neutral': 0.1,\n",
       "    'positive': 0.02}],\n",
       "  'query_response_hallucination': {'entailment': 1.5,\n",
       "   'neutral': 97.8,\n",
       "   'contradiction': 0.7},\n",
       "  'context_response_hallucination': {'entailment': 10.3,\n",
       "   'neutral': 79.7,\n",
       "   'contradiction': 10.0},\n",
       "  'harmful_query': False,\n",
       "  'harmful_context': False,\n",
       "  'harmful_response': False,\n",
       "  'refusal_response': False},\n",
       " 'reference_based_metrics': {'query_response_bertscore': {'precision': [0.8446345925331116],\n",
       "   'recall': [0.8695610761642456],\n",
       "   'f1': [0.8569165468215942],\n",
       "   'hashcode': 'roberta-large_L17_no-idf_version=0.3.12(hug_trans=4.41.0)'},\n",
       "  'query_response_rouge': {'rouge1': [0.125],\n",
       "   'rouge2': [0.0],\n",
       "   'rougeL': [0.125],\n",
       "   'rougeLsum': [0.125]},\n",
       "  'query_response_bleu': {'bleu': 0.0,\n",
       "   'precisions': [0.07692307692307693, 0.0, 0.0, 0.0],\n",
       "   'brevity_penalty': 1.0,\n",
       "   'length_ratio': 3.25,\n",
       "   'translation_length': 13,\n",
       "   'reference_length': 4},\n",
       "  'query_response_bleurt': {'scores': [-1.2369751930236816]},\n",
       "  'query_response_meteor': {'meteor': 0.10204081632653061},\n",
       "  'context_response_bertscore': {'precision': [0.8416609168052673],\n",
       "   'recall': [0.8466401696205139],\n",
       "   'f1': [0.8441432118415833],\n",
       "   'hashcode': 'roberta-large_L17_no-idf_version=0.3.12(hug_trans=4.41.0)'},\n",
       "  'context_response_rouge': {'rouge1': [0.09523809523809525],\n",
       "   'rouge2': [0.0],\n",
       "   'rougeL': [0.09523809523809525],\n",
       "   'rougeLsum': [0.09523809523809525]},\n",
       "  'context_response_bleu': {'bleu': 0.0,\n",
       "   'precisions': [0.07692307692307693, 0.0, 0.0, 0.0],\n",
       "   'brevity_penalty': 1.0,\n",
       "   'length_ratio': 1.625,\n",
       "   'translation_length': 13,\n",
       "   'reference_length': 8},\n",
       "  'context_response_bleurt': {'scores': [-1.3652904033660889]},\n",
       "  'context_response_meteor': {'meteor': 0.05319148936170213}},\n",
       " 'string_similarities': {'query_response_fuzz_q_ratio': 33,\n",
       "  'query_response_fuzz_partial_ratio': 52,\n",
       "  'query_response_fuzz_partial_token_set_ratio': 100,\n",
       "  'query_response_fuzz_partial_token_sort_ratio': 52,\n",
       "  'query_response_fuzz_token_set_ratio': 38,\n",
       "  'query_response_fuzz_token_sort_ratio': 38,\n",
       "  'query_response_levenshtein_distance': 49,\n",
       "  'query_response_bm_25_scores': array([-0.27465307]),\n",
       "  'context_response_fuzz_q_ratio': 40,\n",
       "  'context_response_fuzz_partial_ratio': 40,\n",
       "  'context_response_fuzz_partial_token_set_ratio': 100,\n",
       "  'context_response_fuzz_partial_token_sort_ratio': 50,\n",
       "  'context_response_fuzz_token_set_ratio': 42,\n",
       "  'context_response_fuzz_token_sort_ratio': 42,\n",
       "  'context_response_levenshtein_distance': 47,\n",
       "  'context_response_bm_25_scores': array([-0.27465307])},\n",
       " 'response_text_stats': {'result_flesch_reading_ease': 90.77,\n",
       "  'result_flesch_kincaid_grade': 2.1,\n",
       "  'result_smog_index': 0.0,\n",
       "  'result_coleman_liau_index': 3.82,\n",
       "  'result_automated_readability_index': 2.0,\n",
       "  'result_dale_chall_readability_score': 6.57,\n",
       "  'result_difficult_words': 2,\n",
       "  'result_linsear_write_formula': 2.0,\n",
       "  'result_gunning_fog': 2.4,\n",
       "  'result_text_standard': '1st and 2nd grade',\n",
       "  'result_fernandez_huerta': 122.72,\n",
       "  'result_szigriszt_pazos': 122.96,\n",
       "  'result_gutierrez_polini': 51.88,\n",
       "  'result_crawford': -0.8,\n",
       "  'result_gulpease_index': 95.7,\n",
       "  'result_osman': 89.92}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_all(\n",
    "    \"Everyone is a terrorist\",\n",
    "    [\"Eminem is the white legend\", \"Trump's a bitch\"],\n",
    "    \"There is no answer to that. These questions and context are bad\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py39",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
